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My concerns in this paper are as practical as the designing, testing, and 
revising of courseware to cause the PLATO computer system to become a "teach- 
ing machine" for elementary school mathematics, or as the selection of ef- 
fective strategies for teaching a high school i^athematics course, or as play- 
ing a successful role as a parent. (Indeed, I have present responsibilities 
in all three areas that are exceedingly real and demanding!) That I am led, 
nonetheless, to some brief consideration of rather "scholastic" or "academic" 
matters is, I believe, inherent in the present state of affairs, because some 
of our biggest uncertainties are uncertainties about what sort of thing to 
look for and how to go about looking. 

I. Some Fundamental Questions . Is there, in fact, a prospect of an "applied 
science" of education? I might begin by stating a few of my personal biases, to 
help the reader to discount them. I like the logical structure of mathematics, 
and if I could shape the world according to my wishes , I would give the study 



The research reported here has been supported, in part, by NSF PES 7m 2567, 
NSF PES 67-00223 and US NSF C 723. 



4U 



-2- 



of education a logical structure sirralar to that which matherr.atics has. I 
also like the role of intuition, ingenuity, and insight in the heuristics of 
making mathematical discoveries, and I would like to see the study of education 
share this, also. Finally, though I have less direct personal experience with 
this, I greatly admire the success of physical scientists in creating abstract 
models that very precisely mirror physical reality in matters like force, 
velocity, acceleration, gravitational attraction, and classical electromagnetic 
theory. I would like to witness the creation of similar models for teaching, 
learning, and human information processing. 

But I would NOT like to deceive myself into imagining these attributes for 
educational theory if this is contrary to fact. Precisely there is the rub. 

What are the facts? 

A. What Is to Be Learned? Looking at the knowledge which we want our 
students to acquire, do we know what kind of kr :>wledge that is? Some of it, 
presumably, is information that is coded in commonly-shared symbols, such as 
natural English, or technical refinements of natural English, or the language 
of mathematics, the notations of organic chemistry, and so on. Let's call this 
explicit , language -coded knowledge . 

It often seems that educational settings assume that all knowledge is of 
this type. One infers this from the prominence of books, lectures, and 
multiple-choice tests. There is, however, reason for doubt. 
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Saying "I must make the ball go strai^^hter^' does not necessarily make 
your muscles sti'aighten out your hook iri bowling. Saying "I want a more exciting 
first paragraph^^ doesn't automatically enable you to produce one. What does? 
On present evidence I'd have to say that I'r: not sure. It may be some use of 
explicit language- coded knowledge, but t:;en again it may not be. Most drivers 
develop a sense of when iheir car is foliowfn;' too closely behind the car 
ahead without needing to look at tiieir speeJoneter, and a good mathematician 
can often quickly oketcii out a proof of a theorem without working out - or 
knowing - the details. Is this achieved by some combination of explicit, 
language-coded knowledge, or not? 

Recently, a student, concerned with generalizing 

X + 3 = 3 + X , 

said "That three doesn^t ha"e to be a three," [Davis, 1972]. I suspect this 
language 5 while literally meaningless, could easily be receded into standard 
English - but this use of language shows some signs of slipping away from a 
precise, explicit use of standard English, into something a good deal more 
impressionistic - but very valuable, nonetheless. 

The main point is that one does MOT, at present, knov/ that all of the 
knowledge that we want students to acquire is (or can be) expressed in explicit 
"publicly-shared" common language. Since we do not know this, and even have 
some reason to doubt it , we probably cannot subscribe to a "science" of 
education that implicitly assumes that all knowledge is explicitly stateable 
in some "shared" language. This quite theoretical concern has, unfortunately, 
a great many practical implications, 
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[ Incidentally , I interpret much of the recent study of heuristics by 
Papert and others , and much work in artificial intelligence , as investigations 
of how far we can invade this unknown territory of bowling, creative writing, 
mathematical thinking, chess playing, and so on, by using explicit language- 
coded knowledge. The verdict certainly isn't in, yet. Maybe explicit knowledge 
will take us the whole way, but we can't be sure at present.] 

In any event, as one of my three examples will show, some of what we wish 
our students to acquire is of a different nature. We might speak of non- 
knowledge goals , such as the acquisition of good work habits. 

B. Knowledge About Education . In the preceding section we were concerned 
with the things we want our students to learn. But how about our own knowledge 
of how to help students learn? That is to say^ what can we know about teaching, 
learning, and the var^ious aspects of human information processing and human 
performance? Everything that can be said about student knowledge applies also 
to our professional knowledge about education. 

For one thing, we should notice that there are alternatives to what might 
be called "scientific" knowledge. The laws of harmony from the Haydn - Mozart - 
Beethoven - Brahms period are one instance. A scientific law is precise in its 
description both of cause and of effect. The common-practice-period harmonic 
rules were explicit and precise in terms of "inputs" or "causes" - what one 
should, or should not, do - e.g., don't use parallel fifths. But they were not 
explicit in describing the results that could be obtained » and the rules could 
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be broken - sometimes to excellent effect, as in the last movemen-f: of Beethoven's 
Opus 59 number 1. We cannot similarly choose to violate the law of gravity; 
moreover, given changing expectations of results, the laws of harmony can be 
replaced by a quite different set of admonitions, as in some contemporary 
electronic music; the results may be different, but may still be entirely 
satisfactory. 

Consequently, the laws of harmony give us a concrete example of a useful 
body of knowledge that is not *'scient if ic" knowledge. 

Perhaps closely related to the laws of harmony are what Polanyi has called 
"practitioners' maxims." "Don't smile before Christmas" is one well-known 
example of a practitioner's maxim shared by some teachers. "Don't be afraid 
to show the child that you care about him" is another, shared by a rather 
different group of teachers. 

Scientific generalizations are by no means the only kind of knowledge 
used by teachers, and are probably overshadowed, for most teachers, by some of 
the alternative forms of "knowledge." All of this adds up to the need for a 
more precise kind of analysis: we must go beyond words such as "science" or 
"art," and seek more precise descriptors such as "fully-algorithmic observation 
procedures," "detailed model-based conceptualizations," etc. When we con- 
ceptualize heat as the motion of molecules , we relate it to a model that deals 
also with kinetic energy, the pressure of gases, mean-free-path calculations, 
and so on. This is different, in an important way, from most statistical 
"knowledge." If we have statistical data on the effect of cigarette smoking 
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on the incidence of lung cancer, or the effect of vitamin C on the frequency 
and severity of colds, we remain unsatisfied. This does not go far enough to 
constitute an "answer." Why not: Apparently we feel a need to know, at a 
bio-molecular level, just how the vitamin C is inv olved in chemical reactions 
at cell walls, or what chemicals in cigarette smoke have what effect on DNA 
chemistry. Only at this level would we feel we had an "answer." This 
distinction gives rise to a descriptor like the one mentioned above: "de- 
tailed model-based conceptualization." and it is phrases such as this that are 
needed, rather than merely terms such as "scientific" or "objective." 

The "scientific" study of education is further confronted with the possi- 
bility that many basic concepts have not yet been formulated correctly. "Force," 
"weight," "mass," and "acceleration" are instances of concepts that are just 
right , at least for classical physics. But if you study a classroom full of 
9th graders who are learning algebra, or 13-year-olds practicing the piano, 
it is by no means clear that the usual descriptors and concepts of educational 
theory are sharply and correctly defined. 

For all of these reasons, there is abundant justification for questioning 
how much of our professional understanding of "education" deserves to be called 
"scientific," or even should be cast in the mold of "scientific" thought, and 
in particular there is great justification for fearing that a too-narrow 
definition of appropriate kinds of professional knowledge will prove harmfu l 
to the practical operat ion of educational ventures. 
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Conc^^rns of this type are now being voiced in many quarters. A random 
handful of books and journals from a few shelves in my office provided the 
following : 

One cannot help but question the significance of psychology's 
contribution to the development of effective instructional procedures. 
...upon closer examination, it is evident that [various recent de- 
velopments, including computer-assisted instruction] are not as 

I 

closely linked to psychological research as many might believe. 

...The more serious problem, however, is that psychologists know 
a. great deal about the acquisition of individual facts and skills, but 
very little about how these combi ne to form a meaningful mental 
structure. Effective methods for acquiring skills and facts are im- 
portant, but the major problem is the development of knowledge structures 
that are more than the sum of individual facts. In order to deal 
effectively with educational problems, we need theories that tell us how 
knowledge is represented in memory, how information is retrieved 
from that knowledge structure, how new information is added to the 
structure, and how the system can expand that knowledge structure 
by self-generative processes. [Atkinson, 197^1] 



The topic of educational research is one which is inherently 
controversial. Criticism has been based upon its amount as opposed 
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to the value of iij ruiiLiI.t.-j (Tate, 1950), the lack of productivity 
of its theories (Jacobson, 1971), its domination by psychology 
(Barron, 1972), and its lack of breadth (Taylor, 1973). To this 
list of complaints I would like to add another problem: the over- 
use and perhaps abuse of the linear model by educational researchers. 
It is my belief thdt the educational research community might 
do well to question the present use of the linear model and to 
consider an evaluation of its usefulness. The foundations of 
this problem go deeply into the sociology of knowledge where 
fundamental issues such as just how science progresses (or ought 
to progress) are debated. [Brown, 197 5] 

One reaction to disillusionment, with the linear model could 
be a return to non-quantitative methods such as historical and 
legal analysis, the analysis of logical arguments, participant 
observation, interviews, non-numerical simulations, case studies 
and comparati VG* approaches . Each of these is valid in its own 
context and Is associated with a unique set of strengths and 
weaknesses. [Brown, op.cit.J 

It is not difficult to appreciate why educational researchers 
are sometimes accused of a preoccupation with quantification 
(Mitra, 197U). Such criticism is especially justified when the 
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But ill l.ict, :>rec ly riil^ premise rO'-.{uir^e;:; discussion. If, again, one 
looks closely at .:tadont:s and at schools, ii is eritirely easy to suspect that 
there ar'e important rind, of raiowledge that mav not be of this nature. 

Rapoport and Morvath (1958) refer to Whitehead's notion of 
ttiC hemming in of scientific thought and the tr.reat of impending 

1 0 ru 1 i t y I' ti c a us e 1 1 1 e c u 1 1 ur e "cannot b ur\'i t t : : r o ugh the f r amew or k 
of its own concepts,..*' They '.p'.a- of the rroblenn of assuming that 
a compl^'X fihen.oriK.'non can : e understood by treating it (as the psycho- 
statistical nodel does) as if it can b*^ "broken up... in to a temporal 
chain of evf^nts, all connected by determinate 'causal' relations." 
Such statistical studies of separate variables within the organi- 
zational pi^ocess has, in our view, tested "ungrounded" and often 
meaningless hypotheses and promises sterility in organizatio. and 
administrative education theory in the future . The time is long 
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overdue for another approach in education. The anthropological 
field study approach provides hope for breaking through this con- 
fining framework of present concepts and opening a new area :^or 
educational research. [Lutz and Ramsey, 1974J 

While there are many possible criticisms of educational re- 
search, inconsistency of models is not one of them. It is natural 
to understand why certain disciplines which view the world in 
characteristic ways can afford methodologi*^s which are relatively 
uniform and reflected in the research writings they produce. How- 
ever, because the educational field must address a vast array of 
problems, it seems reasonable to argue that our theoretical and 
methodological tools used be correspondingly broad. Surely edu- 
cational research should be more comparable to the 'liverse re- 
search in engineering rather than the more focused research in 
psychology. The danger of a highly uniform set of methods is 
sterility v/hich precludes crucial problem solving due to the tight- 
ness of the imposition of the linear model paradigm. [Brown, 1975] 

Most passionate, and far-ranging, was a commentary by Hazel Henderson, 
ported by Constance Holden: 

In Henderson's view, corporations are not only central but are 
symbolic of what's wrong with the way we think about things. When 

1 1 
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she speaks of the "Cartesian trip'' she refers to the constellation 
of values and structures that have directed "progress" in this 
century - centralized, hierachical, huge organizational structures, 
the commitment to "big-hang, capital-intensive technology," the belief 
that science and technology are value-free, and over-emphasis 
on linear, objective, reductionist thought - or, dipping into 
psychology, what she regards as reliance on the brain's left- 
hemisphere thinking as opposed to the output of the right hemisphere, 
which is supposed to be the source of spontaneous, intuitive, 
emotional impulses. 

She believes that recognition that all choices are based on 
values has fallen out of this rigid and highly compartmentalized 
system, and that the springing up of public interest groups is 
one of the signs that the system is beginning to crumble because 
of its increasing inefficiency in meeting peoples' needs and the 
growing social costs incurred. [Holden, 1975] 

Very similar arguments, applied to the case of medicine, are presented in 
Carlson (1975). 

The preceding quotes come from a quite literally random handful of books 
and journals. Had I deliberately sought them out » I could have found many, 
and more strongly worded, cautionary statements - for example in position papers 
written by Peter Hilton. .^ald Rising, Burt Kaufman, and others. 
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I'roin all of this I conclude that we must move cautiously and wisely in 
the development of any theoretical understanding of education. There is a 
widespread and legitimate concern that obvious aspects may be treated in 
unsubtle ways, while the less obvious is neglected, with the result 
elusive, complex, and profound qualities may become lost. We have a this 
happen in recent years, and we have had chances to observe how a compulsive 
concern for secondary phenomena, when elevated to a level of policy, can be 
unmistakably harmful. 

For the sake of the record.^ it should be mentioned that critics who 
object to ''the linear model'' do not all mean the same thing. Some refer to 
seeking linear relationships among variables; others object to the assumption 
that every valuable conceptualization is describable in terms of variables; 
still others object to the assumption that one first does research, after 
which one "applies" it, and point out that development often comes first , and 
is followed by research , as in the massive NASA development effort that put 
men on the moon, after which they picked up some rocks and brought them home 
for the purposes of research . This is most directly relevant to education. 
Personally, I would like to see some experimental schools started on the basis 
of developmental work, and carried to the point where they made possible some 
new ventur^es ijn research - perhaps including research into the hypotheses 
that underlie the original developmental work* < 

Let us turn now to some specific examples. 
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II- Motivation and Work Habits , In the fall of 1974, Jody Douglas, an 
anthropologist at the University of Illinois, lived briefly in the homes of 
4 eighth grade students, observing adult/child interactions in the hope of 
getting deeper insight into why some students are motivated to do meticulous 
and virtually perfect work, and to learn lessc " 'y, while other stu- 

dents are apparently content to think about aca .ters in a superficial 

way, to pass in incomplete assignments (or even none at all), and to learn 
lessons imperfectly or worse - 

On the basis of the Douglas observations, it was possible to identify 
two types of family environment. Type A and Type B. The Type A milieu is 
characterized by very frequent adult participation in decision making - 
Type A parents help select the clothes the child will wear to school, 
help decide which parcies the child will attend, help decide how much TV, 
and which programs, the child will watch. In the contrasting Type B |^ 
pattern, the child selects for himself the clothes he will wear to school 
each morning (and may even decide which clothes will be purchased), experiences 
little review (if any) of his social decisions, has relatively unrestricted 
access to TV (and may even have his own personal TV set in his room), and 
comes and goes with few if any restrictions. The expectations are as different 
as the actions: the child brought up in a Type A milieu clearly expects adult 
involvement in all of these decisions, and seems to accept this state of 
affairs. The Type B child does not expect adult "intrusion" into "his affairs, 
and resists any such intrusions. Even if adults are trying to help the child 
in fairly obvious ways, the child may refuse their help, so zealously does 
he defend his independence. 
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There is a difference, also, in what is done for the child. Type A 
parents spend more time supervising their children, but they also spend more 
time chauffeuring their children around, helping with homework assignments, 
etc. Type B parents allow the child to go many places, but mainly via bicycle, 
bus, or a ride with someone else's parents. 

There ait. no major economic diiiureuces between Type A and Type B families 
all were in the over $20,000 per year, and under $60,000 per year, income range, 
and lived in homes in the $40,000 to $90,000 range. All frithers were pro- 
fessional men. Type A mothers did not hold jobs outside of the home, since 
maintaining a Type A milieu was clearly a very demanding job; several Type B 
mothers did hold full-time jobs outside of the home. 

There were no major I.Q. differences among the children; all nad meas 
I.Q. 's over 125. 

There was an absolute differer-.L in ethnic background: all Type A fami 
came from the same ethnic/religious background, and, while not identical, the 
ethnic/religious backgrounds of the Type B families covered only a narrow 
range, quite different from that of the Type A families. 

Remember that the Type A and Tyr>^ B milieu's were aefined from observations 
of home environments . Their defii.iixra did not look at school performance. 

Dut did the 'ype A vs. Type r tinction predict school performance? 
Yes, iL a quite remarkable degree. -ire is one instance: on the first mathe- 
matics test given the year following che Douglas study, there were 22 students 



in the class that Douglas had studied. There was adequate data to allow the 
classification of 5 families as Type A, and 7 families as Type B; for the 
remaining 10 families, there was inadequate data to allow classification - 
but in any event, it is presumed that, with adequate data, the Type A and B 
categories would NOT exhaust the whole range of possibilities; presumably many 
other types of home environments would be identifio/. if enough families were 
to be studied. Now, here are the examination scores on the first mathematics 
test of the following year: 

Grades of students from Type A families: 
97, 97, 93, 91, and 91. 

Grades of students from Type B farrii.:.ie 
73, 67, 66 , 61, 60, 54, and 51. 

Grades of studerrts from unclassified homif^ environments : 

97, 91, 88, 86, 82, 80, 75, ,37, *ith one student absent from 
the exam. 

This is by far the greatest predic^ ve iccess that we have had. : '^hs 

it tend to confirm or deny the notion - e :ation has at last become m 
applied science? 

I 'm not sure . 

Arguing for caution, arguing againsi a t ro-hasty acceptance of education 
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as an applied science, there are some points to be made: first, Douglas is 
an anthropologist, and it has been anthropological methods that have led us 
this far - anthropological methodology being about as remote from "laboratory" 
or "experimental" or "statistical" methods as one usually gets without leaving 
the realm of "science" entirely. Second, the patterns are classified as 
gestalts , not defined by thresholds on a single variable. This is certainly 
NOT the "linear model" that critics were questioning in the earlier quotations. 
Third, this work was based on direct naturalistic observation , and NOT on 
questionnaires or interviews or self -reports by subjects .- methods that have 

thus far characterized a large proportion of the "scientific" study of education, 

2 

much to Its detriment. 

On the pro-science side of the argument, it is clear that next steps 
can be taken to pursue the Douglas study further, and that these next steps 
will include some recognizably "scientific" methodologies, in pursuing such 
questions as: will the distinction in student performance persist over time? 
Pive there areas of study or performance that, unlike mathematics, will not 
show so large a correlation with the Type A/Type B distinction? Is it possible 
to identify a few key variables that reliably define the Type A and Type B 

2 

Cf. also Kline-Graber and Graber's survey of research in human sexuality 
[Kline-Graber and Graber, 1975, pp. 23-27], where questionnaires are rated as 
the least reliable method of data collection, interviews next least reliable, 
observation by trained but relatively inexperienced observers next , with only 
direct observation by trained and experienced observers providing reasonably 
reliable data. Sex studies based on questionnaire and interview data have, 
in the past, painted a confused, contradictory, and quite inacciarate picture 
of human sex behavior. Only recently has direct observation finally begun to 
provide reliable data, from which a less contradictory picture is now emerging. 
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milieus? How well will this distinction hold up when larger numbers of 
families are considered? 

But, again on the side of caution, one important class of further study 
deals with looking for actual detailed changes in student study habits that 
explain the differences in student performance ~ and this, as least at f irst , 
gets us back to direct informal naturalistic observation, a method often 
rejected by many advocates of a "scientific" approach* 

Furthermore, we have come to suspect that much of the difference in 
performance can been be understood in terms of student self -concepts : students 
have a collection of descriptors of themselves, and of external events, and 
criteria for identifying and rejecting mismatches. A student who thinks of 
himself as conscientious would not be content to turn in an English paper 
that he considers "a shoddy piece of work." We also suspect that fairly 
deep introspection is required in order for a perse n to examine his own self- 
concept, and the future self that he expects to become. Added to our distrust 
of questionnaires, this gives us problems when we begin to plan ways to study 
self -concepts . This task resembles psycho-analytic investigations, and, once 
again, we find ourselves on the outside margins of "scientific" methods ♦ 

The goal, of course, is to solve as muc . of the problem as possible: 
to find out what patterns of achievement are associated with wihat patterns of 
childhood milieu. We must distinguish those methodological approaches 
that help us to solve the proh^^em from those methodological approaches which 
impede our reaching a solution. 

X o 
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III » Designing and Evaluating PLATO Courseware > The University of Illinois 

has built, largely from inventions by Donald Bitzer, what is probably the 

3 

largest CAI system in the world, known as the PLATO system. From the point of 
view of a student using PLATO, he has before him, very roughly, a tele'^'i i on 
"•■f. ,nr] in r'" ^r" : *7pewrii r. CActUillv, the "TV screen" is in fact 
Bitzer's p lasma parcel , with writing or drawing in an orange neon glow on a 
black background, and the keyboard is simpler than, and different from, a 
typewriter. ) PLATO is capable of doing nearly anything that a lesson designer 
considers desirable- 

For four y-r^ars we have been engaged in programming courseware for PLATO, 
and during academic year 1975-6 there is underway an official evaluation of 
PLATO and its courseware, as presently used in elementary school reading and 
mathematics, in courses in computer science, accountings a ; gebra , and other 
subjects at' the community college level, and in a wide range of university 
courses, including veterinary medicine, Latin, French, Russian, physics, 
biology, accounting, chemistry, computer science, and many others, I will 
speak only about the elementary school mathematics courseware, and I have 
two points that I want to make: 

1» Concerning the evaluation , if one looks at what has been a 

matter of concern, and at what has not been, one is led to wonder 

how well we are being served by our present attempts to "be scientific," 

3 

For a description of the PLATO system, plsse refer to Nancy Wood. The 
PLATO System , Computer-Based Education Ressrch Laboratory, University of 
Illinois, Urbana, Illinois (1975). 
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2, Concerning the creation of the courseware, what seems to 

be needed is skill in design - and design is, at the r ^t generous 

stimate, only a small per cent "scientific/' being mainly somerhing 
else: luck , judgement, originality, creativity, experience, prej- 
udice, preference, taste, convention, and who knows what else. 
Despite x:Ss fact, design is neither taught nor studied in usual, 
educational circles. It is one of our most conspicuous oraissioms. 
Probably it must be approached via case studies , rather than via 
generalizations . 

First, the evaluation . The elementary math courseware has been under 
construction for roughly three years, during which individual lessons have, 
of coAiTse, been tried out with children. We have called this special testing . 
It serves to help locate certain obscurities, ambiguities, and so on, 
but it provides next to no data about how the lesson will work when a 
student encounters it in its normal place in the curriculum . September, 1975, 
marked our first opportunity to have students progress through the entire 
curriculum according to the intended sequencing procedure, which combines 
computer control with occasional teacher judgements in determining the 
sequential order of lesson material. Thus this is, in a very real sense, our 
first-draft courseware; there has been no opportunity to try to match the 
lesson designer's guesses to the reality of children (except for "special 
testing," which does not address the most importanc questions), and to 
make appropriate adiustments. Whatever our attitude toward "science" as a 
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slogan, we do believe that the courseware MUST be te^teu ^ga^nst reality, and 



adjusted to improve the match. 

Thus we have the spectacle of our first--draft guesswork going through the 
ritual of a ^'summative evaluation/' as if it deserved to be considered a 
final product. There is an Alice-in~Wonderland quality about this entire 
proceeding that is easily understood if one regards this as an instance of 
essentially political behavior, but it cannot be reconciled with a seriously 
"scientific" approach. 

Far from claiming that our first-draft courseware is a finished "package" 
ready for final evaluation, we would prefer guarantees that owr first-draft 
efforts will NOT be released to the world at large until after extensive 
revisions and modifications have been made. 

But there are even deeper reasons for questioning whether the evaluation 
supports the notion of a "scientific" approach, or rather brings it seriously 
into question. Before going on I want to emphasize that my remarks are not 



One important piece of reality concerning the creation of PLATO coiirseware 
is that a very large amount of DEVELOPMENT has to be done BEFORE one can do 
the important RESEARCH - you have to have at least a minimally viable system 
of courseware, in operation in at least a dozen or so classrooms, beforve you 
can observe the effects on teachers and children. The o-cnr-of -context "special 
testing" cannot shed much light on what wiLl happen during day-to-day rouxine 
usage. Few things could be less scientific than to assume that a child's 
10-minute interaction with an oi. t-of-sequence PLATO lesson need only be 
multiplied by 600 to predict the result of routine day-in-and-day-out use 
30 minutes per day, 5 days per week, 40 weeks, per year, in its intended 
sequential order, where carefully building and developing mathematical ideas 
is one of the major goals. This represents one more failure of the "linear" 
model, which assumes that research comes first, and is followed by development. 

' 1 
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criticisms of the evaluators. The evaluation has attracted considerable 
attention, since the over-all effort is a ten-million-dollar ventiare , at a 
time when resources are not generously available, with the result that the 
evaluation has become caught ap in those complex political processes where 
no man can be said to be in control. Probably everyone involved with the | 
evaluation would want it done differently if they could impose their will on 
the venture - although people do not agree on how it should be changed. 
Thus doubts about the evaluation are doubts about the practice of evaluation 
in 1975, with the problem of resource allocation and political processes 
very much a part of the picture. 

For one thing, what proposition is being tested? For some people, the 
proposition says something like this: if, to your regular school curriculum, 
you add daily work on PLATO, using whatever courseware may be available, 
student performance in mathematics, measured by any conveniently available 
tests, will show an improvement. 

I cannot take that proposition seriously. It seems just barely removed 
from an absolutely magical faith in PLATO as an infallible teaching machine. 
Worse, it envisions a use of PLATO which, I feel, should not be allowed to 
occur. I cannot believe that effective educational programs are so chaotically 
random. Surely, it must matter that learning experiences be somewhat 
matched to learner characteristics, and this proposition ignores the whole 
question of whether students ar^ getting appropriately-selected learning 
experiences . 
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We have a different proposition in mind: 

We believe that, if one starts (as we have) with a diverse collection 
of courseware, showing some measure of over-all organization, yet including 
several competing approaches; if one puts this courseware into use in a 
dozen classrooms; if lesson designers and classroom teachers work together 
to find which classroom roles PLATO can play most effectively, and how 
teachers can best use PLATO and PLATO coiorseware; if, on the basis of this 
experience, you make adjustments (and even major revisions) in the course- 
ware; if, as you find how teachers can best use PLATO, you develop a teacher- 
training program; and if you develop, test, and revise appropriate off- 
terminal materials, then you should get to a point where new teachers, 
initially se.ected for general competence, who then go through your teacher 
education program successfully, and who then use PLATO (together with the 
off-terminal materials) in their classrooms in general accord with the scheme 
you have worked out, should find that most of their students are learning more 
mathematics (in a sense which we would make reasonably precise) than had 
formerly been the case. 

That is a proposition I can, and do, believe in. It is NOT, however, 
the proposition that is actually being tested. Indeed, many critics of 
the evaluation reject this proposal as "unscientific,'* arguing that PLATO 
must J^tand alone, without benefit of a teacher-training program. But if 
"science" requires that PLATO stand alone, without being integrated into 
an over-all classroom program, then "science" is requiring that we engage 



in poor- educational i^rac t. Ic*?^ aiid tliat is far too high a price to pay. 

Indotj'-i, w.' h^;a•^! ono of the najor fears of those v;ho resist "being 

more scientific": t:;ie fear that, in order to have a universe that can be 
studied more cleanly, wu shall put educational practice on a Procrustean 
bed, and chop off anyth.ing that doesn't fii: our simple conceptualization • 
We thus get a "reality" that is easier to study, but one that cannot be 
optimized in any e^ffective way. 

t 

In fact, the elome: • iry n:iath coursewcor-e is organized into thi^ee separate 
'^strands," and each strand has been produced by a different team, designed 
according to three different sets of assumptions. 

The so-called "graphs" strand (dealing with variables, functions, graphs, 
negative numbers, etc.), while the least conventional in content, is the 
most conventional in presentation. Straightforward mathematical tasks follow 
the "discovery" format used yn earlier trials of this same material in face- 
to-face teaching. The s^^encing follows a "spiral curriculum" pattern, 
which means that an idea is not completely developed in a short time interval; 
rather, the student cycles back to it, in different forms, so that the idea 
builds up gradually; intuitively, at first, followed by gradual explication 
and further development. Actual experiences are straightforward explorations 
of mathematical situations. 

The "fractions" strand uses a "story" format: for example, in operating 
a pizza stand and filling orders for one- third of a pizza, etc. The sequencing 
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13 fairly iinear, v/iti: .m idea being developoa ir. a relatively short period 
of time; the pre and pest testing is the most fully developed of any strand, 
and certain levels of success must be achieved before a student is allowed 
to go on to new material. 

The "whole nurnberr." strand is the hardest to describe. The ci.u?riculum 
sequencing is very loose, and most experiences are in the form of games, so 
that one gets the impression of a collection of games that may be played in 
any order whatsoever. 

This diversity of approaches creates a real possibility for evolution. 
As classroom experience develops, and revisions are made, the different 
strands will inevitably come to resemble one another more closely, but the 
initial variety means that the evolution may proceed in whatever direction 
classroom experience shows to be most desirable - it is evolution in an 
almost biological sense, where Variation and selection for superiority are 
the two cornerstones. By contrast, if all of the coursewai'-e were organized 
according to a single format and a single educational philosophy, we would 
be largely locked into that single approach. Extensive evolutionary 
modification would be quite unlikely. 

I propose to estimate the vigor and appropriateness of today's **scientif ic" 
approach to evaluation by reviewing some of the things that have been matters 
of concern in the PLATO evaluation, and some of the things that have not been. 

Things that have NOT been matte .' f concern (but should have been ) . 
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We have already mentioned the lack of concern for the fact that only a 
preliminary version, a "first-draft," of the courseware is being evaluated 
as if it were a final product . 

We have also mentioned the lack of concern for wh \ch propositions about 
PLATO are being rested. The conventional wisdom deais so exclusively with 
"complete packages" that PLATO, and its attendant courseware, are automatically 
.construed to constitute a complete package. Teacher training, in this view, 
would be an inappropri ■ contamination of a "scientific" evaluation. 

Also neglf^cted is the fact that PLATO uses a single, very large, time- 
shared computer, and that 800 terminals are located all over town, in schools, 
university classrooms, libraries, offices, and even homes. Every terminal is 
connected to the same computer. PLATO is very popular with students. Thus, 
when it is assumed that a class is a "control" class because there are no 
terminals in that classroom, this assumption is less than safe. Many children 
in "control" classes are, in fact, regular users of PLATO, on evenings and 
weekends, either at a university site, or at their father's office, or 
even at home. A good journalist would have found this out rather quickly; 
the "scientific" approach to evaluation did not raise any questions in this 
area. A control is a control is a control. 

Which child is signed on? PLATO, of course, can't tell. When from 
generosity, extortion, or otherwise, one child acquires another child's "secret 
password," the door is ,ppened to considerable doubt as to which child is 
actually signed on. This has not been seen as a matter requiring study. 
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Who is in the experimental group? Given what has been said above, there 
follows another problem. If some "control'* children are actually on PLATO, 
it is equally true that some children in the '^PLATO" or ''experimental" classes 
^Q"^ ^se PLATO , They trade their turns to other children. 

In two instances, a teacher who teaches two classes is serving as his 
own control: PLATO is used in one class, but not in the other. Of course, 
to the extent that PLATO achieves its results indirectly, by influencing 
teacher behavior, these "control" classes may actually be PLATO experimental 
classes . 

We regard the teacher's role as crucial. Yet this is not receiving very 
careful study. 

In some cases, special extra help, from university volunteers, is being 
provided to non-PLATO students, because teachers perceive these children as 
being denied the advantages of PLATO sessions. This, again, raises questions 
as to what proposition is being studied. Is it the relative advantage of 
human help over machine help? 

The tests used in the evaluation include some widely-used standardised 
tests of mathematical achievement. Some of these tests do not include, among 
the possible answers, the most likely "wrong" answers. E.g, , on one question, 
6 -f- y does not include "3" as one of the possible answers- 

The reading level of the students was not a matter of concern, even though 
the <iiUo- tutor ial aspect of PLATO relies entirely upon reading and writing 
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for computer/ child communicatiDns . 

But most important of all are these two omissions : the trial is not 
seen as a brief time slice in an on-going evolutionary process, where 
continuous observation leads to continuous modification and adjustment of 
the program, and very little attention is focussed on the most important 
question of all: what actually happens when a child sits at a PLATO terminal 
and attempts to work through a mathematics lesson . 

This sort of thing does not seem to fit easily into present-day "scientifi 
conceptualizations of CAI evaluations. 

A fundamental point of disagreement is over the nature of feedback data 
that should be used for program modification. We believe in watching the 
children very closely, and making courseware changes quickly when they seem 
to be needed. Two minutes of observation of one child may give enough data 
to suggest a revision, and a further hour or two of observing a few other 
children may confirm this to the point where we believe the change should be 
implemented - that same day I We are very eager to gather long-term data, 
but we do not think one should spend a year collecting data in order to decide 
what changes to make. Such a time scale is completely wrong. 

Of course, every "omission" listed above could be embraced within a 
scientific study of PLATO - but that is not the question. Rather, ask: 
how real is the danger that the slogan of ''being scientific" will be used 
to justify the process of over-simplifying reality, constraining it to find 
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a ri^iiii rmtary conceptualization, snd i:CToring everything thar doesn't quit* 
seer . ~it. Fr --^ir present vazrz^- pcint , the C:i:?>:;^-: ^'^^err^s very real 



of cciTic ii, in the sense that centred: dnd experimental r ^sses needed to be 
matche, for S.E.S. But one moves frc this important tr::ith to a rather 
dubious proposition; it was felt that a wide range of S.E.S. children should 
be represented in the test population. Why? The modification of courseware 
to api^eal to different student populations sho^ald be a separate task. There 
is something magical in assuming that specific attention to demographic 
modifications is unnecessary; a broad-range success should just automatically 
appear, coming from nowhere. 

It has been strongly suggested that the teachers who use PLATO should 
have been selected randomly, instead of taking volunteers (which is what we 
did). There seem to us to be two major errors in this: we would never tell 
a teacher that he or she had to use PLATO, whether they wanted to or not- - 
that seems to us to be the wrong way to run schools; also, there is again 
the question of what proposition is being tested - we do NOT believe that 
randomly-chosen teachers should be using PLATO, per^haps even against their 
wishes. The only question that makes educational sense to us sees PLATO 
as part of a larger program, including off -terminal materials and special 
teacher training, being used by teachers who want to use it, and who enter 



^.j that w^RE of conce: 



evaluation. 



isly, and correctly, soc ^- 



onomic status of students was a matte? 




the program already competent in trad. 



-Hg skills. 



Once again, a concern for the sloga: 
tort the educational reality in quite har-^ 

There was concern to eliminate the ^z. 
This resulted in trying to match experime • 
school - ignoring the fact that the teach 
extremely different teachers , who were nir 
race, teaching styles, personalities, and 
ignoring the fact that contamination of t'r 
into PLATO when they weren't supposed to 
were side by side. 



'"re scientific" seems to di 



-I.^d geographical variable-" 

i-. control classes in the same 
di rs '^nat resulted consisted of 
- :-.le at all (differing in sex, 

:nal philosophies), and 
"T'oi class by children getting 
liespread since the classrooms 



This pattern of decision-making based cn slogans of "being scientific" 
seems to us ritualistic, a kind of magical tr.:n^ing, an assumption that looking 
carefully at the schools, the teachers, and rrip ^ Mdren i.: unnecessary, that 
instead one should manipulate certain symbols _ t the "geographical variable") 
sanctified by the professional literature, nnd ^is:. ume that one has thereby 
dealt effectively with the reality. To t'^e extent that the slogan of "being 
scientific" is used in this way, it seems seriously detrimental to the Im- 
provement of educatron. 
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Let us turn now to the task of designing PLATO lessons, 

Cr.e attribute of our- design work is that it is theo ry-related . For 
the-ore^'ical reasons we distinguish at present) seven curriculum tasks, as 

fc ^low£ : 

readiness -building 

paradigm creation (cf, Davis, 1972) 

practice of routine skills ("sskills," in the sense of Henkin, 1975) 

developing algorithms or procedures ( ''sub-routines" or "programs" in 
the sense of computer programming or artificial intelligence) 

developing higher-level procedures that select, sequence, or modify 
sub-procedures (cf . Davis , 1975 ) 

practice in non-routine settings (which includes, as a sub-category, 
experience in pattern discovery) 

developing explicit heuristics 

Another attribute of our lesson-design is our use of minute naturalistic 
observation of what happens when a child sits at a PLATO terminal and tries to 
work through our lessons. 

In point of fact wc, do try to be "scientific," in that we seek careful 
explication of whatever we do intuitively, to the fullest extent that we can 
achieve such explicaticr. ; we seek generalizations or propositions that may be 
testable; and we try for objective observation procedures. Unfortunately, 
reality continually confounds this effort. The situation-specific attributes 
of lessons continue to dominate the generalizable aspects. Let me give a few 

1 



e>«ainples, taken from actual observatior notes. Please :;i-rice tna": rhese 
examples cover a lot of ground, -angirzi: in a sezise froir. "ne surLime to the 
ridiculous, and demanding cures .-• ang'!— from trivial correctioi: rf obvious 
design errors, to quite subtle --^r proround modifications, some-zimes even 
requiring the creation of a greater artificial intelligence capHbililr?' with 
PLATO'S programmiTig : 

Example 1. A lesson on open sentences, like 
3 X □ = 12 

etc. (from the Graphs Strand), 

Problem 1. Students choose equations to work on from a list of 37. The 
vast majority of thesr^ require non-integer solutions . At present the curriculm 
makes no check to see that students know something about fractions or decimals, 
and many of their, don't. ^ Students are usually not ready for this lesson - at 
least in its present form. I have watch«d many siradents in this lesson, but I 
have yet to see one for whom this lesison was a user^xil learning exsserience (with 
one possible exception, whom I discuss below). 

Problem 1 could be solved bi many ways: xn~- ^urriciulum checiks could be 
inserted; the problems could be classified into appropriate categories, and 
students could get the easier catEego:rLes first; the car'-^gOT'ies ccnuld be labelled^ 

^Recall that the present year of -lassroom trials is the first toting of all 
of the courseware, with full use of computerized lesson selection. 
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30 a student wo — i know v. at to ::p.er- one or more sarnplat .problems could be 

worked out or. ^ ..TO as i. -dzzraris.Tis — L? sequences coi:l± e designed; or 

an off-PLATO in oductior. the -=Hc-r:"^" could be provide . using hard-copy 

materials tha^ v could create. .rr- ar_y several of tnes mprovement should 
be made . ^ 

Problem 2. The "one possiblr- excrrrtion," mentioned ^ove, was a very 
bright and consc^^intious girl name:: P. , who got to 

IC X Q = 101 

and, unlike the other children, dia ncr ^ive up. However, ?. did not know 
what to do. She had tried 11 and found :.t too large, and 10 and found it too 
small. P. had not yet goxxen to rract:L^iis, neither in the second curriculum nor 
on PLATO. She had some very xTagie?-: ides abour how decimals might work. She 
finally- dbcided to try - tentaxfzjRely - 10. 1. She keyed In 1, then 0, then the 
decimal point. The lesson acr^srr ed the 1 anc the 0. -rrr rejected the decimal 
point. She accidentally pushe:^ SEXT, not meariing t^^. --syid the lesson cycled her 



through a long routine t snow HiaT .10 was toe smsi:^, veri-ch she alrescdy knew; 

7 

there was no vay to ^umr .Uttrac m-; sequeace. rscause she couldn't key 

it in, she couldn't ti^* 10..^ - it would wCTi . At Tiiis point she needed. 



Part of the difficult-' here is trne lessee 3=5i^ers wanted this lesson to 

serve as "practice in a non-rcut_z:e ir^tting-'' S.anie :)f th^ proposed remedies 
would remove the lessen from :nrregory. 

7 

In an earlier version of PLAT?, c .^led PLATC III, the keyr^t included a key 
labelled ''AHA!"; this terminated a -"LLP sequence, signifyir^ that the student 
now saw the difficulty and wanted proceed on his own. This key may need to 
be reinstated. 

J 3 
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and got, some human he^p, frc»n an adult who asked (as ?LATC c^^uld have done) 
"How many tenths in one?** She answered "10'\ Adult: "Ho- nu^v tenths in 
13?" That took nore thought, but P. finally decided '■'^e^■^ were 100. At th:^s 
point she wanted to try Neither she nor the adu- : hac --eaOly noticed a 

line on the screen that said "If yc^u want to put in a fract±on, press f." 
[In rany lessons we have so much writing the kids are leamjzig not to borthsr 
reac^ng most of it.] Finally soTr.e-:x):ry noriced this line, so P. pushed f and a 
sma — arrow appeared _n a raised position, apparently ^eadr ro accept a 
numer^ator. P. tried zd key in iOl. but the lesson acctsptea only 10, arparent- 
ly nz. accepting three-digit numerators. At this pcin- P. :zDuld think X3f no 
way to try out 10.1. She had to appeal tc an adult., who showed her how to key 
in a correct answer (as a mixed number). 

The solution here is trivially onnrlous , zrut doe-^ nesed to be made, and making 
the necessary changes can dramaticall-y alter the impa v. of this lesson cmr :::hildren. 

Example 2. In order to help chi-ldz^en know 'nax %irneemat:ics to use in '^rd 
problems," there 13 a sequence dea liix^ wirth th^ "Tiiear L r!igs^ cr the various oper- 
ational (in the Whoie Numbers Strand). The lefs: :ti in ..uestion is intended to 
establish two meanings for multiplication: the "'repeated adrxitian" meaning 
(7 + "^+7=3x7, etc.), and the "replication" meaning. The student is given 
a "rubber stamp" and a rectangle representing "a piece of paper." Each tiiii: he 
stamps on the paoer , he marks two dots. 

The lesson iissuined that a chilL? wc.i.l':' • mr i •- an esserrially meffiiingless 
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pattern, such as either 



or 




after which the student was expected to wirilre a descrip^^jnL. of the dcts ±n an 
addition story 

. 2-r2 + 2 + 2 + 2 + 2 = 12 

and then in a mult iplicat ion story, 

6 X 2 = 12 . 

One child I watched this week - a minority boy whc seemed :rat uhs' defeatred 
oy sizhool in general, and always on the verge of giving up, - in any sir::ool- 
relsted situation ~ had made a rather careful pattern that I inagine he vas 
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rather proud of: 



He tried to describe bis creation by the addition sentence 
6 + 6 = 12 , 

but the lesson rejected this. He couldn't understand what was wrong. An adult 
finally intervened and rold hir: that the lesson probably expected the answer 

2 + 1-^2+2 + 2 + 2 = 12 , 
so he typed this in, and it was accepted. Then the lesson asked him to describe 
the dot situation by means of a multiplication statement, and he was completely 
stumped. After he had sat there for some time, an adult asked him [following 
George Polya's advice about the "inventor's paradox" heuristic] how much 7 
2*s would add up to. The boy answered correctly, but again could not make a 
multiplication sentence; the adult asked how much 15 2's would add up to. 
The boy answered correctly, and was now able to make the relevant multiplication 
sentences in each case. [David Page suggested the strategy of moving to larger 
numbers in order to clarify the algebraic structure of a situation.] 

Without adult intervention, this lesson failed, but it could easily be 
fixed up. 



Solutions: 1) The lesson could be more accepting of student responses ♦ 
Given the student response 

6 + 6 = 12 , 

the lesson could ask: **Does your arrangement of stamps suggest 6 + 6?'* 
(or some such thing), thereafter saying: Alright. Now suppose we ignore how 
the stamps were placed, but try to count the dots just from the fact that there 
are 2 dots on each stamp. Would this be a good description: 

2+2+2+2+2+2+2= 14? 
[This would lead to answer "No, because, there are 6 stamp marks on the paper, 
and the addition sentence uses 7 2's", which might be developed via some 
multiple choice questions - the point here is to get students answerin g , with- 
out making them guess what was in the mind of the lesson designer .] The lesson 
would then go on to establish 

2 + 2 + 2+2+2+2= 12, etc. 

2) Or one could use a somewhat simplified version of solution 1, making more 
use of PLATO responses like: "Well, 6 + 6 = 12 is true, but what I had in mind 
was 

2+2+2+2+2+2= 12." 

3) Or one could change the lesson by having PLATO work the first example, by 
way of illustration* This is probably a major difference cognitively , for the 
cognitive Task now is merely' one of copying an example, which does NOT necessarily 
require the same kind of thoughtful analysis that 1 or 2 would. Solution 3 is 
probably cognitively inferior to 1 or 2, but would be better than just leaving 
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Example 3: Another boy - again, a defeated-appearing minority child who 
doesn't seem to hold high hopes for school - working in the lesson where you 
move a turtle to his food, in order to learn to plot points in the first 
quadrant in Cartesian coordinates. 

This boy hardly reads any writing on the screen, and may have a very low 
reading level. He strikes keys so rapidly - and sometimes almost randomly - 
that the lesson can't keep up with him. It is continually trying to flash a 
message from 8 or 10 key-presses back. 

It seemed to both observers that the boy was not building up any patterns 
of mathematical structures in his head. On the contrary, he was approaching 
the lesson in the spirit of those psychological experiments where you hit 
some pattern of buttons to make certain lights go out. At his fastest and 
most frantic rate, I doubt that he knew the labels on most of the keys he hit. 
He finally worked out the following pattern: if (say), a message appeared on 
the screen saying: 



he would press 3 [which actually had no effect but he didn't realize that]. 



[Even this much order was achieved only aften an adult got the boy to stop his 
wild hitting of the keys.] 

I cannot say that this boy was learning mathematics. It is entirely 



II 



The food is at (3, 4) 



then press 4 [again, no effect], then push the 
moving, then push the a, | key until the turtle 




Solution: I think the be;.-: solution for this type of child is intervention 
by a helpful and supportive ad _t, though oirviousiy triere are some improvements 
that could also be made in thi. lesson. [Hun: ice the cmier lying problem here.] 

If a student is allowed to make an unboimn-ftd sequ^ce of errors (which 
no teacher would allow in face-to-face teaching)-, tbe student can become hope- 
lessly mired down in confusion, making no prr:sr*"=^^. [In such a case, the 
computer reports this situation to the teacher, out one obviously prefers to 
have PLATO solve its own problems wherever possiiile, since demands on the 
teacher's attention are always too numerous in xiTiy classroom.] But, on the 
other hand, if N errors (for seme value of Wf- arrrivate some automatic help 
sequence, then students need nor analyze pT»t±:Z»sraE at: all - they can merely hit 
keys randomly until the help sequence appesrs-. 

Good lesson design obviously has to :rry to operate between these two 
extremes. The reader can probably easily irrw»ent modifications that will 
eliminate the difficulty in the present insitrannH . 

One could even make a syston for doing sc — but we feel that systems should 
NOT be so evident to the learner that sessions tnn PLATO become entirely routine. 
[Variety is the spice of CAI.] 

Example 4. A girl, Cecily, working en the '"egg-dropper" lesson. This 
lesson is essentially a readiuess building experience for the addition and 
subtraction of signed numbers. A 'Victim" - a man standing on a number line - 
is shown at, say 9 ("negative 9"). Higher on the screen, a helicopter hovers 
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directly over 3. The student (who has voluntarily selected this form of the 
lesson, presumably because ho enjoys pr^^tendinp^ to drop eggs on people's heads), 
by keying in 6, would move the helicopter 6 units in the negative direction, 
thereupon dropping an egg (and scoring a direct hit). The screen looks some- 
what like this: 




I — I — I — I — I — I — I — I I I I — I— I — I — \—\ — I — I — h> 

'8 '7 "6 ^5 '4 "3 "2 "l 0 ^1 ^2 ^3 \ ^5 ^6 ^ ""s "^Q 

Instead of keying in 6, Cecily keyed in 's. [The observer's notes record 
that this was the second time Cecily had missed by one unit.] 

Using random numbers, the lesson then offered the following sequence of 
problems : 

1. victim at "? 

helicopter at 5 
correct answer i 
Cecily's ansv/er 0 



missed by 1 point 



2. victim at "lO 
helicopter at ~13 
correct answer 3 
Cecily's answer 3 

3. victim at 13 
helicopter at 8 



direct hit! 



EKLC 



-no- 



id 

ERIC 



victim at 
helicopter at 
correct answer 8 
Cecily's answer 5 



missed! 



victim at 13 

helicopter at 10 

correct answer 3 

Cecily's answer "^3 



direct hit! 



victim at 
helicopter at 
correct answer 
Cecily's answer 



11 
"ll 
0 

"O [sic] 



direct hit! 



victim at 12 

helicopter at 6 

correct answer 6 

Cecily's answer "5 



missed by one unit! 



victim at 7 

helicopter at 1 

correct answer 8 

Cecily's answer "^5 



missed! 



victim at 
helicopter at 
correct answer 
Cecily's answer 



missed by one unit! 
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10. 



11. 



12, 



victim at 
helicopter at 
correct answer 
Cecily * s answer 



victim at 
helicopter at 
correct answer 
Cec i ly * s answer 

victim at 
helicopter at 
correct answer 
Cecily's answer 



11 

"8 
"6 



4 

"6 



0 

"3 



Missed, and, for the first time 
Cecily spoke aloud; in fact, she 
screamed: "Oh! It's minus seven!" 



direct hit! 



missed! 



Whatever is going on in Cecily's head is fairly complicated, but you can 
explain a great deal of it if you assume the following: 

1. Cecily was not relying primarily on the numerals , but rather on the 
picture of the number line. 

2. Her basic error was to count the d ivision marks , rather than the 
number of unit intervals » 



8 

The author was assisted in both the observations and in their analysis by 
Sharon IXigdale, a PL^TO lesson designer. 
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[This by itself explains the errors in problems 1, 7, and 9, and her (wrong 
verbal remark in problem 10.] 

3. However, she did have a correct procediare available in her repertoire; 
the choice of the correct or incorrect procedures was somewhat random, but was 
influenced by the cognitive difficulty of that particular task: the greater the 
difficulty, the more likely a wrong choice (cf. Davis, 1971-2 B). This may 
explain why Cecily made a correct choice on problem 2 ( 10 and 13 being easy 
for her to deal with?), on problem 5 (again, 10 and 13), and on problem 11 
( U and 6 being easy for her?). 

It may have been the case that getting the n^^ problem correct 
increased the likelihood of selecting the correct sub-procedure for problem 
n + 1, especially if the (n + l)st problem was cognitively similar to the 
n^^ problem. This might explain the correct answer on problem 3 (where the 
numbers were 13 and 8). 

There is something interesting to be studied here. For the immediate 
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task of improving the effectiveness of this lesson, however, there are many 
possible ways to proceed. Here is one: since we do believe in the phenomenon 
of regression under conditions of cogr*'tive overload, perhaps the lesson should 
at first use only positive integers, with the victim on the right, so that 
the correct answer is also positive, and one has only the pattern of straight- 
forward addition (adding the helicopter's move to its original position): 

victim at 12 
helicopter at 8 
correct answer 4 

In order to make sure that students are (sooner-or-later ) using the 
numerals instead of the picture , after a while, suppress the written scale on 
the picture. (Following David Page's suggestion, you could also go to larger 
niambers, thus compelling a use of the subtraction algorithm, as in: 

victim at 1492 
helicopter at 866 
correct move ? 

but this might be counterproductive, since it may represent a premature 
appearance of a further complicating factor. ) 

The continuation of the lesson in this fashion follows fairly obviously. 
IV . Summary 

The fear is often ex'^ressed that "being scientific," as a slogan, is 
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nowdays frequently leading to oversimplifications of reality, and thus to poor 
educational practice. Our experience, unfortunately, tends to confirm this. 

On the opposite side, when we try to approach the task of lesson designing 
in a systematic, theory -based fashion, we encounter two major obstacles: 

i) The situation-specific aspects of lessons seem to dominate the 
generalizable aspects ; and ii) The scale of feedback is far different from 
the usual "scientific" scale: instead of studying a thousand students for a 
year (although this is important), we get our most valuable design ideas from 
studying a few students for a few minutes, and then implementing some changges. 

We are left with the belief that, in 1975, too much attention is focusing 
on "being scientific,'" and far too lit1:l-e attention is focusing on design* 
Design in education has much in common with design in women's dresses 5 or in 
furniture, or in architecture. The effective study of design probably needs 
to build on case studies more than on generalizations. 

Careful naturalistic observation has received far less attention than it 
deserves. Before we get too involved in phenomena, methodology, or theories, 
we need to take a good look at what's going on > The methods of good journalist 
can help, at this stage, more than the subtler methods of statisticians. 

It might help us to move out of this dilemma if we dropped "science" as 
a slogan-word, and instead tried to use more precise and emotionally-neutral 
descriptors, such as these: 

Does the observation method depend upon a rather fully -structured 
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explicit a priori procedure, or does it depend upon observer choice in 
selecting observations? 

Is the interpretation of observations related to a theory, or not? 
If so, to what theory? 

Are descriptors applied to observed gestalts , or to sharply-defined 
single variables? 

The slogan of "b^ing Hcienfific" does KDT , at present, seem 
genuinely helpful, fe can^ and must do betrar. 
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